The Lagrange method and SAO with bounds on the dual variables

We consider the minimization of f 0 ′ +1 ≤ j ≤ m and x ∈ X , where X is compact. For any λ ∈ R m , let x(λ) be a minimizer of the Lagrange function L(x, λ) = f 0 (x) + Σ m j=1 λ j f j (x), x ∈ X , and let φ be the dual function φ(λ) = L(x(λ), λ), λ ∈ R m. Assuming only that the functions f j are continuous, it has been proved that, if x(λ) is unique, then φ has the derivatives dφ(λ)/dλ j = f j (x(λ)), 1 ≤ j ≤ m. Thus we deduce that, if φ(λ *) is the greatest value of φ(λ), λ ∈ R m , subject to λ j ≥ 0, m ′ +1 ≤ j ≤ m, and if x(λ *) is unique, then x = x(λ *) is the solution of the given problem. These properties are illustrated by an example with n = 2 and m ′ = m = 1. The given problem may have no feasible point, however, and then φ may not be bounded above. Therefore the bounded dual method adds the condition λ ∞ ≤ Λ for some prescribed Λ > 0, and we let λ * be the new maximizer of φ. We find that, if x(λ *) is unique, then now it minimizes the function Ψ(x), x ∈ X , which is f 0 (x) plus Λ times the sum of moduli of constraint violations at x. The term SAO stands for Sequential Approximate Optimization. An outermost iteration makes quadratic approximations to the functions f j , 0 ≤ j ≤ m, with first order accuracy at x (k) , say, where k is the iteration number. Then, using the approximations instead of the original functions, the bounded dual method is applied, giving a new λ * and a unique x(λ *). The choice of x (k+1) can depend on Ψ(x (k)) and on the new Ψ(x(λ *)). Thus our theory suggests some useful developments of SAO.